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Abstract 

This paper deals with the analytic continuation of holomorphic automorphic forms on a 
(Hermitian) Lie group G . We prove that for any discrete subgroup T of G there always exists a 
non-trivial holomorphic automorphic form, i.e., there exists a T-spherical unitary highest weight 
representation of G . Holomorphic automorphic forms have the property that they analytically 
extend to holomorphic functions on a complex Ol'shanskh semigroup S C Gc ■ As an application 
we prove that the bounded holomorphic functions on T\S C T\Gc separate the points. 



Introduction 

Let G be a connected Lie group sitting in its complexification Gc • Let r < G be any 
discrete subgroup. Then T acts freely and properly discontinuously on Gc and the quotient 
r\Gc is a complex manifold. It is reasonable to ask which such homogeneous complex manifolds 
are Stein. If G is nilpotent, then T\Gc is Stein by a theorem of Gilligan and Huckleberry (cf. 
[GiHu78]). We call a Lie algebra g weakly elliptic, if all operators &d(X) , X G fl, have imaginary 
spectrum and note that all nilpotent and many solvable Lie algebras are weakly elliptic. Then 
Loeb has shown in [Lo84] that T\Gc is Stein whenever the Lie algebra g of G is weakly elliptic. 
But if G is semisimple and T is a lattice, then a theorem of Barth and Otte (cf. [BaOt73]) 
implies that the only holomorphic functions on T\Gc are the constants; in particular T\Gc is 
not Stein. Therefore in general the G x G-biinvariant complex manifold Gc is too large for 
r\Gc carrying a rich complex structure. In this paper we will identify an open G x G-invariant 
Stein domain S C Gc for which we prove that there is a rich supply of automorphic forms on 
r\G which holomorphically extend to T\S . In an important example we will prove that T\S is 
Stein. 

Let us illustrate the situation for the example G = SU(p, q) . Then G can be defined as 
the invariance group of a hermitian form (•, -) p ^ q on C" , n = p + q, and we have Gc = Sl(n, C) . 
The contraction semigroup of the form (•, -) p>q 

S:={g E G C - (g-v,g.v) p>q < (v,v) Ptq for all v e C™, 

is easily seen to be an open G x G-invariant domain in Gc, a so-called complex Ol'shanskii 
semigroup. In general one can define complex Ol'shanskii semigroups for every connected Lie 
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group, whenever its Lie algebra admits an open convex weakly elliptic Ad (G) -invariant cone 
7^ W C g. If g is simple, this means that g is either hermitian or compact, but we also have 
many non-reductive examples as the Jacobi-algebra t)„ xsp(n, R) with f)„ the 2n+l -dimensional 
Heisenberg algebra (cf. [Ne99a] and Example 1.3 below). The complex Ol'shanskh semigroup 
associated to G and W is given by S — GExp(iW). 

In [Ne98, Ne99b] Neeb settled a conjecture of Gindikin by showing that all complex 
Ol'shanskh semigroups and their symmetric space analogues are Stein manifolds. Let now T < G 
be any discrete subgroup. In this paper we investigate the quotients T\S . These manifolds were 
first considered by the first author for G = Sl(2, R) in the context of Hardy spaces on T\S (cf. 
[Ach99]). 

Let us in the following assume that Ad(G) is closed in Aut(jj) and that W is pointed, 
which both are very natural assumptions in our context. Then one of our main results is: 

Theorem A. (cf. 4-V If T < G is a discrete subgroup of G , then: 

(i) Hol(r\S) separates the points of T\S . 

(ii) IfT<Gis cocompact, then the bounded holomorphic functions on T\S separate the points. 
In particular, the Caratheodory semimetric on T\S is a metric. 

We construct holomorphic functions on T\S with techniques from representation theory. 

Let tCj denote a compactly embedded Cartan subalgebra of g. If (tt\,H\) is a unitary 
highest weight representation of G with highest weight A G it* , then one has a holomorphic 
extension to a representation of the complex Ol'shanskh semigroup S which has G as some sort 
of Shilov boundary (cf. [Ne99a, Ch. XIV]). We obtain holomorphic functions on T\S by taking 
matrix-coefficients with L-fixed hyperfunction vectors; i.e., we take 

:T\S—>C, Ts (tt\(s).v, rj) , 

with v G Hx and n G W^" a T-fixed element. Here "H^ denotes the Frechet G-module 
of hyperfunction vectors of (ir\,H\) , the strong antidual of the analytic vectors Ji^. Note 
that for v being a highest weight vector the function 6 VtV is an analytic continuation of the 
(anti)holomorphic automorphic form 6 VtV \q (cf. [B066]). 

Our considerations from above show that one gets many holomorphic functions on T\S 
provided T -spherical unitary highest weight representations {■n\ 1 TL\) of G do exist. Our central 
result is as follows: 

Theorem B. (cf. 3.8 - 3.11) Let F be an arbitrary discrete subgroup of G and assume that G 
has compact center. Denote by A C it* the spectrum of the L 1 -Bergman space B 1 (S) on S . 
Then the following assertions hold: 

(i) For all X G A and v G TL^ the Poincare series of v 

p {"»)■= J2 nx ^- v 

converges in the module of hyperfunction vectors Ti.^ to a T -fixed element. 

(ii) For all but finitely many A G A we have P(v\) ^ 0, where v x is a highest weight vector of 
(tta,Wa)- 

Part (i) is obtained from integral geometric observations in the Bergman space B 1 (S) 
together with the Plancherel theorem for B 2 (S) (cf. [Kr98a]). 

To get a feeling for the contents of (ii) we explain it for the example G = SU(1, 1) . Write 

D = G/K for the open unit disc and write elements of T as 7 = ( ° 7 ^ 7 ) • Then can 
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be naturally be identified with Hol(D) and we have 

(*) PM(z):=5> 7 z + d 7 ) A (zeD). 

Here the parameter —A ranges in the integers with A > 3. The idea for the proof lies in an 
analytic continuation of P(v\) in the parameter A together with a clever argument that this 
analytic continuation is non-zero. 

Statements related to (ii) were also proved by Siegel (cf. [Fr83]). He considers scalar valued 
Poincare series (this means that the minimal if -type of (jr\,7i\) is one-dimensional) and proves 
that for a given T there always exists a if -finite vector v such that P(v) ^ 0. Siegel's proof 
relies heavily on the fact that D is a bounded domain while in our approach this is immaterial 
(for the Jacobi group D is unbounded). Further our argument does not need that the Poincare 
series is scalar valued. Finally, Siegel's method does not tell for which if -finite vector v one has 
P(v) ^ while our techniques yield P(v\) ^ for the distinguished highest weight vectors v\ . 
We would also like to mention that (ii) partly solves an open problem posed by Wells and Wolf 
(cf. [WeWo79]; see also [WaWo83]). 

In Chapter 4 we show that the analytically continued automorphic functions Q v ^ vanish 
at infinity: 

Theorem C. (cf. 4-6) If T < G is cocompact and v e then we have 

lim 9 V n (s) = 0. 

ser\s 

It is our impression that this result in combination with Theorem A might yield the Stein 
property of T\S in the cocompact case (cf. Conjecture 5.3 below for a more detailed discussion). 

For G = Sl(2, R) the existence of a non-vanishing holomorphic cusp form (the discriminant) 
allows to prove the following: 

Theorem D. (cf 5.1 - 5.2) For G = S1(2,R) and all subgroups T < S1(2,Z) the quotients T\S 
are Stein. 

The authors would like to thank Karl-Hermann Neeb for his very useful remarks. 



1. Complex Ol'shanskii semigroups and their representations 

In this first section we introduce complex Ol'shanskii semigroups, which may be thought of 
as complex Lie subsemigroups of complex Lie groups. Then we recall the basic facts concerning 
their holomorphic representations and the characterization of the irreducible ones by highest 
weight representations. The concepts will be illustrated with some importnat examples (cf. 
Example 1.3 and Remark 1.7). 



Complex Ol'shanskii semigroups 

Throughout g denotes a finite dimensional real Lie algebra. The concept of a complex 
Ol'shanskii semigroup is closely related to the concept of elliptic elements in a real Lie algebra 
g. Recall that an clement X E g is called elliptic if &dX is semsimple with purely imaginary 
spectrum. Accordingly we call a subset W C g elliptic if all its elements are elliptic. 



4 



March 15, 2001 



Definition 1.1. (Complex Ol'shanskii semigroups, cf. [Ne99a, Ch. XI]) Let ^ W C g be an 
open convex elliptic cone and W its closure. Let G, respectively Gc , be the simply connected Lie 
groups associated to g, respectively gc, and set Gi: = (expg) C Gc- Then Lawson's Theorem 
says that the subset r Gl (VF): — G\ exp(iW) is a closed subsemigroup of Gc and the polar map 

G lX W^r Gl (W), { g ,X)^gexp(iX), 

is a homeomorphism. 

Now the universal covering semigroup Y~{W): — r Gl (VT) has a similar structure. We can 
lift the exponential function exp: g + iW — > r Gl (W) to an exponential mapping Exp: g + iW — > 
T~(W) with Exp(0) = 1 and thus obtain a polar map G x W -> rg(W), (#,X) i-> gExp(iA) 
which is a homeomorphism. 

If G is any connected Lie group associated to g , then m (G) is a discrete central subgroup 
of T~(W) and we obtain a covering homomorphism T~(W) — > r G (FF): = rg(W)/7ri(G) . It is 
easy to see that there is also a polar map G x W — > Fg(W / ), (5, A) 1— ► gExp(iA), which is a 
homeomorphism. The semigroups of the type Tg(W) are called complex Ol'shanskii semigroups. 

The subset T G (W) C L G (I4 / ) is an open semigroup carrying a complex manifold structure 
such that the multiplication is holomorphic. Moreover there is an involution on Tq(W) given by 

*:r G (W) -> r G (VF), a = 9 Exp(iX) ~ s* = Exp^A).?" 1 , 

being antiholomorphic on Tg(W). Thus both T G (H / ) and r G (T4 / ) are involutive semigroups. ■ 

From now on we denote by S an open complex Ol'shanskii semigroup Tg(W) and by S 
its "closure" T G (W). 

If [ is a subalgebra of a Lie algebra g, then we write Inn (I) C Aut(g) for the subalgebra 
of the automorphism group Aut(g) of g which is generated by the elements e adx , X 6 I. If 
[ = g, then we also write Inn(g) instead of Inn g (g) . 

A subalgebra o C g is said to be compactly embedded, if Inn g (a) is relatively compact in 
Aut(g) . Note that a subalgebra is compactly embedded if and only if it is elliptic. 

Note that if a real Lie algebra admits a non-empty open elliptic convex cone, then there 
exists a compactly embedded Cartan subalgebra t C g (cf. [Ne99a, Th. VII. 1.8]). To step 
further we first need some terminology concerning Lie algebras with compactly embedded Cartan 
subalgebras. 

Definition 1.2. (cf. [Ne99a, Ch. VII]) Let g be a finite dimensional Lie algebra over K. with 
compactly embedded Cartan subalgebra t. 

(a) Associated to the Cartan subalgebra tc in the complexification gc there is a root decompo- 
sition as follows. For a linear functional a £ t c we set 

0g:={Xeg c :(Vr etc) [Y,X] = a(Y)X} 

and write A: = {a G t c \{0}:gg ^ {0}} for the set of roots. Then g c = tc©0 Q(EA gc, a(t) C iR 
for all a G A , and gg = g^ Q , where X X denotes complex conjugation on gc with respect 
to g. 

(b) Let r denote the radical of g and note that there is a t-invariant Levi decomposition g = r x s . 
Let t be a maximal compactly embedded subalgebra of g containing t. Then a root a is said 
to be compact if gg C 6c and non-compact otherwise. We write A^ for the set of compact roots 
and A„ for the non-compact ones. 

(c) A positive system A + of roots is a subset of A for which there exists a regular element 
Xq G it* with A+:= {a G A:a(Xo) > 0}. We call a positive system t-adapted if the set 
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A+:= A„ n A + is invariant under the Weyl group Wf. = Af Inn£i ( t )(t)/Z Ilm£i ( t )(t) acting on t. 
Recall that there exists a {-adapted positive system if and only if 3 (3 (6)) = t. In this case we 
say is quasihermitian. In this case it is easy to see that s is quasihermitian, too, and so all 
simple ideals of s are either compact or hermitian. 
(d) We associate to A+ the convex cones 



C m i n := cone{i[X a ,X a ]:X a E flg, a E A^t}, 

and C max : = («A+)* = {X E t: (Va E A+) ia(X) > 0}. Note that both C min and C max are 
closed convex cones in t. 

(e) Write pt : £J —* t for the orthogonal projection along [t, 0] and set Ox:= Inn(fl).X for the 
adjoint orbit through X E Q. We define the minimal and maximal cone associated to A+ by 

W min : ={XE g:pi(Ox) C C min } and W/ max : ={le fl:pt(0x) C C max } 

and note that both cones are convex, closed and Inn(g) -invariant. ■ 

Example 1.3. (a) (Hermitian case) Let q be a semisimple Lie algebra with Cartan decomposi- 
tion q — 6 ©p. Then q is called hermitian if g is simple and 3(6) ^ {0}. Hermitian Lie algebras 
are classified; the complete list is as follows (cf. [Hcl78, p. 518]): 

su(p,q) sp(n,M) so*(2n) so(2,n) e 6( _ 14) e 7( _ 25 )- 

That g is hermitian implies in particular that there is a compactly embedded Cartan subalgebra 
t C £ of fl and that 3(8) = RX is one dimensional (cf. [Hel78, Ch. VIII]). Since X is Inn B (t)- 
fixed, the prescription 

A+:= {a E A:a(iX ) > 0} 

defines a J-adapted system of positive non-compact roots. Hermitian Lie algebras admit pointed 
convex Inn(g) -invariant closed cones with non-empty elliptic interior. One of them, the minimal 
cone, has a quite simple description: 



W min = Inn( ).K+X o . 
Its trace with the Cartan algebra t is given by 



C m in = cone{-«a:o! E At} 
where a E it denotes the coroot of a (cf. [HiNe93, Ch. 7] for all that). 

(b) (Compact case) (cf. [HiNe93, Ch. 7]) Let f be a compact semisimple Lie algebra and set 
g: = t ©R (direct Lie algebra sum). If (•,•) denotes an Inn(6)-innner product on 6, then the 
prescription 

W: = {(X, t) E q: (X, X) < t 2 , t > 0} 

defines an open elliptic Inn(g) -invariant convex cone g. The most prominent example herof is 
= u(n) with 

W: = {X E u(n): iX positive definite}. 

On the group level with G — Xj(n) and Gc — Gl(n,C) the OPshanskh semigroup associated to 
—W is the unit ball in Gc : 



S = Gcxp(-iW) = {g E Gl(n,C): \\g\\ < 1} 
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with \\g\\ the spectral norm. 

(c) (Mixed case - The Jacobi algebra) There are interesting examples of Lie algebras admitting 
invariant elliptic convex cones which are neither scmisimple nor solvable (see Ex. 4.9(b) for a 
non-nilpotent solvable example). One of them is the Jacobi algebra 

g: = t)Sp(n, R): = h n x sp(n, R) 

with t) n = R©R 2 ™ the (2n+ 1) -dimensional Heisenberg algebra. Then 3(g) = i(\) n ) = R is one 
dimensional and a compactly embedded Cartan subalgebra is given by t = 3(g) © t s with t s a 
compactly embedded Cartan subalgebra of Sp(n, R) . For the Jacobi algebra g the maximal cone 
has an easy description, namely 

W max = R+ © M 2 " © W max , s 

with Wmax.s a maximal cone in Sp(n, R). For more details on the Jacobi algebra we refer to 
[KNO01, Sect. II] and [Ne99a, p. 400ff, p. 700]. For another mixed example see also Example 
4.9(c) below. ■ 



Holomorphic representations 

Definition 1.4. Let TL be a Hilbert space and B(TC) the space of bounded operators on it. 
A holomorphic representation (ir, Ti) of a complex Ol'shanskii semigroup S is a non-degenerate 
holomorphic semigroup representation ir: S — > B(H) such that n is a vector-valued holomorphic 
map satisfying tt(s*) = n(s)* for all s G S . ■ 

Now we take a closer look at the irreducible holomorphic representations of S . They are 
obtained by analytic continuation of unitary highest weight representations of G . 

From now on we assume that g contains a compactly embedded Cartan subalgebra t C g 
and that there exists a non-empty open elliptic convex cone W C g. Then in view of [Ne99a, 
Th. VII. 3. 8], there exists a ^-adapted positive system A+ such that 

Wmin CFC W max 

holds, W^ ax is elliptic, W m i n nt = C m j n and W max nt = C max . Recall that every elliptic Ad(G)- 
invariant cone W C g can be reconstructed by its intersection with t, i.e., W — Ad(G).C with 
C:=W nt. 

Definition 1.5. (Highest weight modules, cf. [Ne99a, Ch. X]) Let A+ be a positive system. 

(a) For a g c -module V and (3 G tj we write V := {v e V : (VA G ic)X.v = (3(X)v} for the 
weight space of weight [3 and Vv = {/3- V 13 ^ {0}} for the set of weights of V. 

(b) Let V be a gc -module and v G , u^0,a tc -weight vector. We say that v is a primitive 
element of V (with respect to A + ) if gg.w = {0} holds for all a G A + . 

(c) A gc -module V is called a highest weight module with highest weight A (with respect to 
A+ ) if it is generated by a primitive element of weight A . 

(d) Let G be a connected Lie group with Lie algebra g . We write T for the analytic subgroup 
of G corresponding to t. Let (n, TL) be a unitary representation of G . A vector v G H is called 
T -finite if it is contained in a finite dimensional T-invariant subspace. We write TL T,U1 for the 
space of analytic T-finite vectors. 

(c) An irreducible unitary representation (tt, TL) of G is called a highest weight representation 
with respect to A+ and highest weight A G it* if TL T ' U) is a highest weight module for gc with 
respect to A + and highest weight A. We write HW(G, A + ) C it* for the set of highest weights 
corresponding to unitary highest weight representations of G with respect to A+ . ■ 
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The interplay between irreducible holomorphic representations of S and unitary highest 
weight representations of G is described in the following lemma. Recall that a cone W in a real 
vector space V is called pointed if W contains no affine lines. 

The central ideas in the following theorem go back to Ol'shanskii (cf. [0182]) Stanton (cf. 
[St86]). 

Theorem 1.6. Let S = Yq{W) be a complex Ol'shanskii semigroup and A+ be a t-adapted 
positive system with C m i n Q C C C max ■ Suppose that W is pointed. 

(1) If (tt,H) is an irreducible holomorphic representation of S, then (n, H.) extends to a 
strongly continuous representation of S , also denoted by (ir, TL) , such that tt\q is a uniquely 
determined unitary highest weight representation of G with respect to A + . 

(2) Conversely, if (n\,H\) is a unitary highest weight representation of G with respect to A+ , 
then (tt\,H\) extends to a uniquely determined strongly continuous representation of S , 
whose restriction to S is holomorphic and irreducible. 

Proof. This follows from [Ne99a, Th. XI. 4. 8] together with its following remark. ■ 
We assume in the following that the cone W C g is pointed. 

Remark 1.7. In the special case of the Jacobi group 

G: = HSp(n, M.) = H n x Sp(n, R) 

(a group associated to the Jacobi algebra F)Sp(n, R); cf. Example 1.3(c)) it is interesting to sec 
what the unitary highest weight representations of G are. They are given by 

ct m ® ir x , s 

where is an extended metaplectic representation of G and n\. s is a unitary highest weight 
representation of Sp(n, R) extended trivially to the Heisenberg group H n (cf. [Ne99a, Sect. X.3]). 
All these representations extend to holomorphic representations of the maximal open complex 
Ol'shankii semigroup Tc(int Wmax) • ■ 



2. The manifolds T\S and some concepts from complex analysis 

Let S = Tg(W) = GExp(iW) be an open complex Ol'shanskii semigroup and r < G a 
discrete subgroup. In view of the polar decomposition of S (cf. Definition 1.1), the group T acts 
via 

r x s — > s, (7, s) h-> 7s , 

freely and properly discontinuously on S . Thus the quotient T\S is Hausdorff and carries a 
complex structure which is induced from the quotient map S — > T\S, s i-> Vs. 



Stein manifolds 

Since one of the main objectives of this paper is the investigation of the structure of the 
complex manifolds T\S , especially whether they are Stein, we briefly recall here some facts 
concerning Stein manifolds. 

If M is a complex manifold, then we write Hol(M) for the space of holomorphic functions 
on M . We endow Hol(M) with the topology of compact convergence on M . Note that Hol(M) 
is a Frechet space provided M is second countable. 
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Definition 2.1. (Stein manifolds) Let M be a second countable complex manifold. Then M 
is called a Stein manifold, if the following two axioms are satisfied 
(Stl) The points of M are separated by Hol(M) . 

(St2) The manifold M is holomorphically convex, i.e., for each compact subset K C M the 
holomorphically convex hull 

K:={ze M: (V/ e Hol(M))|/(*)| < su VweK \f(w)\} 

is compact. ■ 

Theorem 2.2. (Graucrt) For a second countable complex manifold M , the following asser- 
tions are equivalent: 

(1) The manifold M is Stein. 

(2) There exists a strictly plurisubharmonic exhaustion function <p:M^M., i.e., <p is strictly 
plurisubharmonic and for all r el the set <f~ 1 (] — oo,r]) is a compact subset of M . 

Proof. [H673, Th. 5.2.10]. ■ 

Theorem 2.3. If E — ► M is a holomorphic principal bundle, such that the fiber and the 
base M are Stein, then E is Stein. 

Proof. [MaMo60, Th. 4]. ■ 

Theorem 2.4. (Neeb) Let S be a complex 01' shanskii semigroup. Then there exists a G xG- 
invariant strictly plurisubharmonic smooth function S ^ M. + with 

lim ip(s n ) = oo 

for all seS\S. 

Proof. [Ne98, Lemma 5.11]. ■ 



Invariant Hilbert spaces 

We conclude this section by introducing the concept of an invariant Hilbert space of 
holomorphic functions. 

Definition 2.5. Let M be a complex manifold and G a group acting on M via 

M x G -> M, (m, g) h-> m.g , 

by holomorphic automorphisms. Then a Hilbert space H C Hol(M) is called a G -invariant 
Hilbert space of holomorphic functions on M if the following two axioms are satisfied: 

(IH1) The inclusion map Tt Hol(M) is continuous. 
(IH2) We have a unitary representation 

p:G^U(H), (p(g).f)(z):= f(z.g). m 

Let (p,H) be an invariant Hilbert space. Then by (IH1) all point evaluations H — > 
C, / i— > f(z) are continuous. Hence there exists for every z £ M an element K z e TL such 
that (f,K z ) = f(z) for all / e H. The function 

K:M x M C, (z,w) ^ K(z,w):= (K W ,K Z ) , 

is holomorphic in the first, antiholomorphic in the second variable and G-invariant, i.e., we have 
K{z.g,w.g) — K(z,w) for all z,w £ M , g E G. We call the function K the reproducing kernel 
of (p, TL) . For further information on reproducing kernel Hilbert spaces we refer to [Ne99a, Ch. 
I-IV]. 
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3. Constructing r-spherical representations 

In this section we give a construction of T -spherical highest weight representations for 
arbitrary discrete subgroups T < G. It was an idea of Godemcnt to obtain T -spherical rep- 
resentations by averaging matrix coefficients of integrable representations of G (cf. [B066]). In 
our setup however it is much more convenient to use the Bergman space B 1 (S) of integrable 
holomorphic functions on S instead of L 1 (G). This is because we can naturally realize highest 
weight representations with sufficiently large parameter in the Hilbcrt space B 2 (S) (cf. [Kr98a]). 



Hyperfunction vectors for unitary representations 
Definition 3.1. Let G be a Lie group and TL a Hilbert space. 

(a) For a unitary representation (n, TL) of G we denote by TL°° and TL" the space of all smooth, 
respectively analytic vectors of (n, TL) . The corresponding strong antiduals are denoted by Hr°° 
and TL~" and their elements are called distribution, resp. hyperfunction vectors (see [KN097, 
Appendix] for the definition of the topology of TL"). Note that there is a natural chain of 
continuous inclusions 

The natural extension of (71", TL) to a representation on the space of hyperfunction vectors is 
denoted by (n^" ,TL^") and given explicitly by 

(Tr-"(g).v,v) := {v^ig-^.v). 

Note that ,TL~ U ) is a continuous representation of G (cf. [KN097, App.]). 

(b) Let r C G be a closed subgroup. For a unitary representation (tt,TL) of G we write (TL~") r 
for the set of all those elements v 6 TL~ U satisfying ^"{^.u = v for all 7 6 T. The unitary 
representation (tt,TL) is called V -spherical if there exist a cyclic vector v 6 (H~ w ) r . ■ 

Remark 3.2. Even though in general the topology on the space of analytic vectors is hard to 
deal with, one has a quite explicit picture for holomorphic representations of complex Ol'shanskh 
semigroups. Let G be a connected Lie group and (tt, TL) a unitary representation of G which 
has an extension to a holomorphic representation of some complex Ol'shanskh semigroup S — 
G Exp(ijy) . Then we have for each X e iW 

U u = (J 7r(Exp(iX)).W, 
t>o 

and the topology on TL^ is the finest locally convex topology on H u making for all i > the 
maps TL — » TL", v ^> 7r(Exp(tA)).v continuous (cf. [KN097, Appendix]). Note that we have 
ir(S).TL- u C TL U and that the action of S on is given by 

{-k^^ (s).v,v) :— (u,tt(s*).v). ■ 

A Lie group G is called a (CA)-Lie group ((CA)=closed adjoint) if Ad(G) is closed in 
Aut(g) . Note that most Lie groups are (CA)-Lie groups; in particular, all reductive and nilpotent 
Lie groups have the (CA)-property (cf. [Ne99a, VII.1.13-VII.1.15]). 
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Lemma 3.3. Let (n\,H\) be a unitary highest weight representation of the (CA)-Lie group G 
with highest weight A G i'miC^^. Then there exists a X G iC with Spcc(dT:\(X)) C —j3 — No 
for some (3 e R. Let (v n ) ne m be an orthonormal basis of H\ consisting of ic -weight vectors. 
Then for all t > there exists a constant C t > such that 

oo 

(Vw G W A ) £ I^A(Exp(tX)). W ,t; n }| < C t \\w\\. 

n=l 

Proof. Let V\ denote the tc-weights of H\. Then V\ C A — No[A+] as (n\,H\) is a highest 
weight module with highest weight A. 

Since C C int C max we have a(iY) > for all a G A+ and Y G C. Moreover C is 
We -invariant so that we can find an element Y <E C such that a(iY) > for all a G A + . Since 
G is a (CA)-Lie group the group T/Z(G) is compact (cf. [Ne99a, Cor. VII. 1.5]) and hence we 
can choose X e iC such that a(iX) G N for all a G A+ . This proves Spec(dTT\(X)) C -/? - No 
for some /?: = -A(X) G K. 

Let Ha = ©^Lo^a^ ™ be the orthogonal decomposition of TL\ into c?7TA(^)-eigenspaces. 
Set m„: = dimH^" and note that there exists an TV G N such that 

(Vn G N) m n <(n + l) N 

(cf [Ne99a, Lemma X.4.9]). Then (u„) neN = («")i<j<m„,n€N with G W^"" for all 
1 < 3 < m n ■ 

Take now v — J2 n c ™ w ™ = n c j v 7 = 7TA(Exp(iX)).u; for some u> G TL\ and t > 0. Then 
w = J2j n e*(" +/3 )c"v™ and we have 

OC OO oo 

El c «l-E E l c "l-E E \c]\e^e~^ 

n—1 n—1 l<j<m n n—1 l<j<m n 

oo ]_ OO 2 

^(E E is"i 2e2t(n+/J) )'(E E e- 2t ^y 

n—1 l<j<m n n—1 l<j<m n 

oo i 

= \\w\\(^m n e- 2t ^y 

n=l 

oo i 

KM^l + n^e-^+^y <C t \\w\\ 

n=l 

with C t := (jln=i( l + n ) Ne ~ 2t{n+0) ) h < oo- ■ 



Bergman spaces 

If G is a locally compact group, then we denote by /lig a left Haar measure on G. 

Let 5 1 = r C ;(W /r ) be a complex CTshanskh semigroup and H < G a unimodular subgroup. 
Recall from [Kr98a, Sect. II] that there exists an S'-invariant positive measure fJ,ms on H\S . 
Denote by Hol(S) ff the space of left i7-invariant holomorphic functions on S . For each 1 < 
p < oo we write || • || p for the norm of the Banach space L P (H\S, Hh\s) ■ For 1 < p < oo we 
now define 

^(tf\S):={/GHol(S)":||/|| p <oo} 
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and note that B P (H\S) is a Banach space of holomorphic functions on S (cf. [Kr98a, Prop. 
II.4] ) ; if p = 2, then B 2 (H\S) is even a Hilbert space. If H = T is a discrete subgroup of 
G, then we have Hol(5) r = Howrys') and we call B 2 (T\S) the Bergman space of the complex 
manifold T\S . 

Proposition 3.4. Let S be a complex Ol'shanskii semigroup and H < G a unimodular 
subgroup. Then the following assertion holds: 

(i) For all compact subsets Q C S there exists a constant Cq > such that 



(V/ e B\S))(Vs e Q) / \f(hs)\ d m {h) < C Q \\fh 

JH 



holds. 
(ii) The linear map 



I H :B\S) - B\H\S), f i ► f H ; f H (s) = f f(hs) d m 

JH 



(h), 



is defined and a contraction of Banach spaces. 

Proof. (i) Let Q C S be a compact subset. W.l.o.g. we may assume that there exists an open 
relatively compact subset U C S such that Q C U C U C S and that U = (H\HU) x iJ holds. 
Thus we may assume that there exists a compact subset Kr Q H such that U = (H\HU) x Kh ■ 
It follows as in [Kr98a, Prop. II.4(i)] that there exists a constant cq > such that 

(VseQ) |/(*)| < cq / |/(«)| d» s (u) 
Ju 

holds for all / G Hol(5) . For each / e Hol(5) and g e G we set f g (s) = f(gs). Note that 
f g e B 1 (S) provided / G B 1 (S) since /i s is G x G-biinvariant (cf. [Kr98a, Sect. II]). Thus we 
get for all / e B 1 (S) and seQ that 

f \f(hs)\ d^ H (h) = f \f h (s)\ du H (h) <c Q f f \f h (u)\ d^ s (u) d^ H (h) 

JH JH JHJU 

< C Q / d lJ-s(u) dfi H (h) 

JH JU 

= C Q / \f(hu)\ dy, H (h) dp s (u) 

JU JH 



= c Q / / \f(h 2 hiu)\ dfx H (h 2 ) duni^hx) dn H \ s (Hu) 

JH\HU JK H JH 

<c Q p H (K„) / / \f(hu)\ d(j, H {h) dfi H \ s (Hu) 

J H\HU JH 

< c Q p H {K H ) f \f(s)\ dp s ( s ) < CqUW,, 



where Cq — cqUh(Kh) ■ This proves (i). 

(ii) Let / e B 1 (S). Then the inequality in (i) implies that f H E Ho\(S) H . It remains to show 
that || f H || i < || /||i. But this follows from 

ll/*Hi = / \f"(s)\ d^ H \ s = I I f(hs) du H (h) dn H \ S {Hs) 

JH\S JH\S JH 



IH\S JH 

completing the proof of (ii). 



< f f \f(hs)\ d^ H (h) dLi H \s(Hs) = 

JH\S JH 
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Let (n,H) be a unitary representation of a Lie group G and (ir* 7 H*) its dual representa- 
tion. We denote by (n, B 2 (TC)) the unitary representation of G x G on Hilbert-Schmidt operators 
on H given by Tr(gi,g 2 ).A — ir(gi)Air(g 2 )~ 1 for gi,g 2 G G and A e B 2 (TL) . Recall that the 
scalar product on B 2 (H) is given by (A, B) = tr(AB*) for all A,Be B 2 {H) . Further we have 
a natural equivalence 

(ir®ir*,H®H*) -> (n,B 2 (H)), v® (-,w) ^ P VtW ; P v , w (u):= (u,w)v. 

Let (R® L,Hol(S)) denote the right-left regular representation of G x G on Hol(5), i.e., 
we have 

(Vffi.sa G G)(V/ G Hol(5))(Vs G 5) ((fl® L)(g ug2 ).f)(s) = /(g^sgi). 

Recall that for a unitary highest weight representation (ir\,H\) all operators n\(s), seS, are 
of trace class (cf. [Ne99a, Th. XI.6.1]). Further the prescription 

(tta, B 2 (Hx)) - (i2 ® L, Hol(5)), ^ h- /i(s) - tr(A7r A (s)). 

gives us a G x G-equivariant realization of B 2 {TL\) as G x G-invariant Hilbert space of holo- 
morphic functions on 5 (cf. Definition 2.5) 

Theorem 3.5. Assume that G is a (CA)-Lie group and that Z = Z(G) is compact. If S is a 
complex Ol'shanskii semigroup, then we have a G x G-equivariant isomorphism of Hilbert spaces 

AeA (^,5 2 (H A )) - (R®L,B 2 (S)), B 2 {Ux)3A^^b(X)f\, 

with 6(A) > for all A e A and 

A:= {A G HW{G, A+)niC*:\ + p€ i int C* uin , X + 2p G i int(G n t+)*}, 
w/iere p: = \ E Q eA+ a and i+: = i X e t: ( Va e A+)ia(X) > 0} . 

Proof. This is a special case of [Kr98a, Thm. IV. 5]. ■ 

For the rest of this section we assume now that G is a (CA)-Lie group, Z(G) is compact. 
We denote by K the analytic subgroup of G corresponding to t. Note that K is compact by 
our assumptions on G. 

Lemma 3.6. We have 2A C A. Further for all A e 2A and a?/ T -finite analytic vectors 
w 6 W^'^ we have a continuous G-equivariant inclusion mapping 

with fp (s) = (n\(s).v, w) , s G S . 

Proof. We first show that 2A C A. Recall the description of A in Theorem 3.5. Let A G A. 
Then p G —iC* nin implies that 

2 A + p = 2(A + p) - p e i int G* in + iG* in C i int G* in . 

Thus 2A is a parameter of the holomorphic discrete series and in particular we have 2A G 
HW(G, A+) (cf. [Ne99a, Sect. XII.5] for all that). Finally we have 

2A + 2p = (A + 2p) + X G i int(G n t+)* + iC* C i int(G n t+)* 
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since A G iC* and C*C(Cfl t+)* . This concludes the proof of 2A C A. 

Let now A £ A and write v\ , respectively v 2X , for a normalized highest weight vector in 
H\, respectively H 2 \ ■ Denote by H\®H\ the G-module where G acts diagonally. Then the 
prescription v 2 \ i— > v x ® v\ gives rise to an equivariant embedding of H 2 \ into H\®H\ (for 
more details see the proof of [HiKr99, Th. 3.2.1] where a similar situation is dealt with). In the 
sequel we assume that TL 2 \ Q H\®H\ . Note that H^x" Q H\ ® H\ so that by linearity we may 
assume that w = w\ ® w 2 for some Wi,w 2 £ H\ . 

Let (v n ) 

n£N be an orthonornial basis of 7Ya • Then (v n %)n,meN 

constitutes an or- 

thonormal basis of TL\®TL\. Hence v £ H 2X can be written as v = J2 n m c n, m v n ® v m with 
IMI 2 = E„,m l c «,™| 2 - Lct X £ iC be as in Lemma 3.3. Then {X,X) £ i(C x C) meets the 
same assumptions as X in Lemma 3.3 for the unitary highest weight representation ir x ® tt\ of 
G x G. Hence if v = ir 2X (Exp(tX)) .u for some u £ H 2 \ then J2 n m l c «,»nl < G t \\u\\ by Lemma 
3.3. Now the Cauchy-Schwarz inequality together with Theorem 3.5 gives 

/ I/p^( s )I d Vs(s) < \ c n,m\ / \{('*\{s)®irx(s)).(v n ®v m ),w 1 ®W2)\ dusis) 

^ s n,m ^ s 

<^2\c n , m \ J \(Trx(s).v n ,wi}(nx(s).v m ,w 2 )\ dns(s) 

<V|c„, m |( / |(tt a (s).?;„,wi)| 2 dns(s)) 2 ( / \{ir\(s).v m , w 2 )\ 2 dfi s (s) ) 2 

= ^yll Wl H • IK II l c ™> m - C *IHI 

with c t : = j^y||wi|| • ||w 2 || < oo. Thus we sec that for all t > the mapping 

n$ x ^B\S), v ~ <S>(Exp(tX).v) , 
is continuous. In view of the definition of the topology of H 2X , this proves the continuity of $ .■ 



Poincare series 

Now we are going to construct L -spherical highest weight representations for generic 
parameters in 2A. 

Definition 3.7. Let (ir,H) be a unitary highest weight representation for G. For v £ H we 
call 

P(v): = ^7r( 7 ).w 
the Poincare series of v, provided P(v) exists in ri~ u , i.e., if 

H w — > C, w i— > ^2{ n {l)- v t w ) i 

is a continuous antilinear functional. Note that P(v) £ (W~ w ) r . ■ 
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Proposition 3.8. Let (n\,H\) be a unitary highest weight representation of G with A 6 2A. 
Then for all w € W[' w the Poincare series P(w) exists, and defines a V -fixed hyperfunction 
vector. Moreover, for all v G H x the series {P(w),v) — X) 7 er( 7r (7)- u; ' w ) converges absolutely. 

Proof. Let w e H T X ' U ,u£^ and set 

f v :S— ► C, s ^ (ttx{s).v,w). 

By Lemma 3.6 we have /„ € B 1 (S) and so fl e B^S) wi * h fv 

converging absolutely on all 

compact subsets QC5, i.e., 

(3.1) (3C Q > 0)(V* £ Q)(V/ e S 1 ^)) X! 1/(^)1 ^ C oH/lli 

(cf. Proposition 3.4). Let X e iC and set s t : = Exp(L¥) for all t > 0. Then Lemma 3.6 asserts 
in particular that for all t > there exists a constant C t > such that 

(3-2) (V^Ha) ll/ 7r ,( 5t ).Ji<C'tH. 

By the definition of the topology on , we only have to check the continuity of the maps 

Tix^C, v~(w x (8 t ).v,P(w)). 

Fix now t > and choose Q such that s: = st e Q. Then (3.1) and (3.2) imply that 

\{n x (s t ).v,P(w))\ = \ ^2(TTx(s t ).v,7rx(-y).w)\ 

< \{^\{s)^x(s).v,nx{'y).w}\ = Y l/^( s ).«(7 _1 s)l 

< CqII/7ta(s).vIIi < Ibll- 

This proves the proposition. ■ 

Now we are going to show that the Poincare series of the highest weight vector P(v\) is 
non-zero for almost all parameters A G 2A. For that we have to recall some facts concernig the 
generalized Harish-Chandra decomposition. Our source of reference is [Ne99a, Sect. XII. 1]. 

We denote by K the analytic subgroup of G corresponding to t . Note that K is compact 
by our assumptions on G . Let Kc denote the universal complexification of K and note that 
K C Kc ■ By Gc we denote the universal complexification of G . 

We define subalgebras of 0c by 

and note that p^ 1 are abelian since W was assumed to be pointed. Further we have gc = 
p~ ®t c ©p+. 

Let P^ denote the analytic subgroups of Gc corresponding to p^ 1 . Assume for the moment 
that G C Gc- Then the multiplication mapping 

P~ x K c x P+ -► P-K C P + , (p-,k,p+) ^ p-kp+. 

is biholomorphic and we have GCSm P~KcP + ■ 

For the general case we can use some standard covering theory to lift the results from above: 
We obtain a complex manifold P~KcP + with a biholomorphic map P~ x Kc x P + — ► P~KcP + 
and the inclusion chain from before also lifts: GCS^ P~KcP + ■ 

We denote by re: 5 — > the middle projection. Set to = tn[t, 6] and note that t = to©3(€) • 
Accordingly we have t* = tg © 3(6)* . 
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Proposition 3.9. Let X £ Hq be dominant integral for AjJ" . 

(i) There exists a lattice C such that 

A + (r 3({) niC*)\{0}C2A. 

(ii) Fix ( £ HW(G, A+) n ((r 3 ( {) n iC*)\{0}) and set A„: = A +nC- TAen i/iere exists an 
No £ N smc/i f/iai A„ £ 2 A for all n > N , neN. Further there exists an neff, n> No 
such that 

(P(v Xn ),v Xn }^0, 
where v\ n is a normalized highest weight vector for (7r An , TL\ n ) . 

Proof. (i) This follows from the structure of the set of unitary highest weights (cf. [Kr99, 
Sect. IV]) together with our explicit description of A. 

(ii) From the definition of A it is clear that there exists an No £ N such that A„ £ 2A for all 
n > N , n £ N. Define N>at : = Nfl [No, oo[. We consider the function 

^:N>Ar -» C, (P(v Xn ),v Xn ) = $^<7rA B (7)-«A„,t>A„>- 

Note that the defining series converges absolutely by Proposition 3.8. 

If (tt\, H x ) is any unitary highest weight representation of G, then F(X): = sp&n{Tr(K).v x } 
is a finite dimensional irreducible representation of K. Hence the representation (ir x ,F(\)) of 
K naturally extends to a holomorphic representation of Kc- By [Kr98b, Prop. 11.20] we now 
have (tt x (s).v x , v\) = (it x (k(s)).v x ,v x ) for all A £ HW(G, A+) and s £ S, and so 

(3-3) (VneN) F(n) = ^(7r A „(«( 7 )).i; A „,i; A „). 

Let n fc := ©„ eA + flg > : = © Q eA- Sc be tne complex conjugate and N k , N k , the 

k k 

respective analytic subgroups of Kc . Then we have the Bruhat decomposition 

A: |J W k wT c N k , 
weWt 

where w £ Kc represents w . Fix neN and consider the function 

f:Kc^C, k i-» (n Xn {k).v Xn ,v Xn ). 

The fact that Ao is regular implies that / vanishes on all cells N k wTcN k with w ^ 1. Write 
an element k £ N k TcN k as k — n(k)t{k)n{k) and note that f(k) = f(t(k)) = t(k) Xn . Thus we 
have 

<«' /<*>-{ V "?S^ 

The set {<(k(7)) £ Tc'-j £ T} defines a countable family (£j)jeN C Tc- Then (3.3) and 
(3.4) give that 

OO 

3=1 

Now it follows from Lemma 3.10 below that F(n) ^ for at least one n £ N proving the 
proposition. ■ 
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Lemma 3.10. Let V be a finite dimensional real vector space, V* its dual and Vc = V + iV its 
complexification. Let C C V* be a convex cone and T C V* be a lattice. Suppose that C P\T ^ 
and that there exists a sequence (z n ) ne ^ in Vc such that 

new 

is defined by absolutely convergent series. Then F ^ . 

Proof. The assertion of the lemma easily reduces to the one dimensional case, i.e., V = M. 
By rescaling we may assume that F = Z. Then C = K or C = ±]0, oof and taking a subcone of 
C we, up to sign, may assume that C =]0, oof. Then we have Zn]0, oo[= N and 



oo 



e 

n=l 

with En=i e mR °( z ") < oo for all m £ N. Hence Re(z n ) > for only finitely many n £ N and 

x : = sup„ eN {Re z n : n £ N} < oo. 
Further F extends to a bounded analytic function 

oo 

/:]l,oo[-C, A^^e Az ". 

n=l 

To obtain a contradiction, assume that F = 0. Then / = by [KrOl, App. A]. 

W.l.o.g. we may assume that Re z„ = xo exactly for 1 < n < N for some N £ N and that 
z\ = Zj exactly for 1 < j < k , k < N . Then dominated convergence gives that 



lim e~ Azi ( Y pXZ3 ) =°- 

A ^°° j=N+l 

Choose A > such that |e~ Azi (J^jZn+i eXzj ) < \ for a11 A > A o- Then / = implies that 

N 1 

(VA>A ) \k+ J2 e tXlm{z i- Zl) \ < -. 

j=k+i 

Since the one parameter subsemigroup 

{( e ^im( 2fe+1 - 2l ) 5 _ _ _ ^ e i\im( ZN -z^. A > Ao | 

in (S 1 ) N ~ k contains elements arbitary close to the identity 1 £ (S 1 )^ - *, we arrive at a 
contradiction, proving the lemma. ■ 

We now summarize the main results of this section. 

Theorem 3.11. Let S = Tq{W) be a complex Ol'shanskii semigroup with G a (CA)-Lie 
group and Z(G) compact. Let U be any open cone in iC* . Then for every discrete subgroup 
r < G there exist a T -spherical contractive unitary highest weight representation (ir\,H\) of G 
with A £ U and compact kernel. 

Proof. Note that non-zero T-fixcd hyperfunction vectors for a unitary highest weight repre- 
sentation (ir\,H\) are cyclic, since is dense in (cf. [HiKr99, Lemma 6.2.1]). Therefore 
it follows from Proposition 3.8 and Proposition 3.9 (ii) that there exists a T-spherical unitary 
highest weight representation of G. Moreover the representations belonging to a parameter 
A £ 2A are contractive since A C iC* (cf. [Kr99, Lemma IV. 13]). The remaining statement 
of the theorem now follows from Proposition 3.9(h) together with [Ne99a, Lemma X.4.7, Th. 
X.4.81. ■ 
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4. Point separation and vansihing at infinity 

In this section we construct rich classes of holomorphic functions on T\S . For T < G 
cocompact we construct a class of point separating holomorphic functions on T\S which vanish 
at infinity. For arbitray T we will show that Hol(r\S) separates the points. 



Compact G-orbits in the hyperfunctions 

Lemma 4.1. Let s a = Exp(X ) with X a G iC . Then the following assertions hold: 

(i) Ss Q C GExp(conv(>V { .X ) +iC')G. 

(ii) There exists an X 6 iC such that 

Ss Q C GExp(X + iC)G. 

Proof. (i) For Z e iC set F z ,x - = conv(W e .(Z + X )) + iC . Then [Ne97, Prop. V.l] shows 
that 

(VZ e iC) Exp(Z)G Exp(Xo) C G Exp(F z ,x )G. 

Now we have 

W t .(Z + X ) C W t .Z + W t .X Q C Wt.Xo +iCC conv(W { .X ) + iC 
for all Z e iC. Hence (i) follows from S = GExp(iG)G. 

(ii) Let <c denote the conal order on the convex cone iC , i.e., for X, Y G iC one has X <c Y 
if and only if X e Y + iC . Since conv(W^.X n ) is compact, we therefore find an element Y E iC 
such that Y <c couv(Wt.X ) , i.e., conv(W{.X ) C Y + iC . In view of (i), this proves (ii). ■ 

Lemma 4.2. Let (ir,H) be a holomorphic representation of S . Suppose that T\G is compact 
and let v r e (H^ u ') r . Then the following assertions hold: 

(i) The orbit 7r~ w (G).-u r is compact in H~ u ■ 

(ii) For all X a £ iC the set 7r~" ( Exp(X ))7r~ w (G)."u r is compact in U. 

Proof. (i) Recall from Definition 3.1 that the representation (7r -w ,W -w ) of G is continuous, 
i.e., the map 

G x W- w - ft—, (g,v)~n-"(g).v, 

is continuous. Thus ir~ UJ (G).v r is compact by the cocompactness of T in G. 

(ii) Recall the definition of the topology on (cf. Remark 3.2). Being the strong dual of the 

analytic vectors K u , the topology on is the finest locally convex topology for which all 

maps 

H^^H, w^7r-"(Exp(tX )).w, t > 0, 

become continuous. Hence for all compact subsets K C Tt~ u , the set 7r~ w (Exp(X )).-ftT is 
compact in H. In view of (i), this proves (ii). ■ 
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Realizing T-spherical representations in holomorphic functions 

Throughout this subsection (n,Tt) denotes a holomorphic representation of a complex 
Ol'shanskii semigroup. We assume that (n,H) is T-spherical for an arbitrary discrete subgroup 

r<G. 

Fix now a cyclic element v r G (TL~ U ) T . By [KN097, App.] the mapping 
(4.1) r:H^Uol(T\S), v ^ f v ; f v (Ts): = (n(s).v, v r ) , 

is injective, continuous and G-equi variant, i.e., the map r gives us a realization of Tt as a G- 
invariant Hilbert space of holomorphic functions on T\S (cf. Definition 2.5). In the sequel we 
assume that TL C Hol(r\5) . Recall from Section 2 that there is a reproducing kernel K for TL. 

Lemma 4.3. Let (tt,TL) be a T -spherical holomorphic representation of S realized in the 
holomorphic functions on T\S via the realization (4.1). If K is the reproducing kernel of TL, 
then Kp s = n(s*).v r , seS, and 

(Vs, t G S) K(Ts, Tt) = (ir(t*).v r , ir(s*).v r ). 

Proof. This is analogous to the proof of [HiKrOl, Th. 4.1.1]. ■ 

Proposition 4.4. Assume that T\G is compact and let (ir, 7i) be a Y -spherical holomorphic 
contraction representation of S . Then the following assertions hold: 

(i) Fix X G iC and set s := Exp(X ). Then, the reproducing kernel K of H is bounded 
when restricted to T\Ss x T\5s . In particular, there exists a constant C — C(s n ) such 
that 

(Vse5 So ) \\K T s\\<C. 

(ii) All analytic vectors Ji^ are bounded holomorphic functions on T\S . 

Proof. (i) By Lemma 4.3 we have K(Ts,Tt) = (n(t* ).v r , tt(s*).v t ). Since \K(z,w)\ 2 < 
K{z, z)K(w, w) for all z, w e T\S (cf. [Nc99a, Ch. I]), we have 

^Pz.wer\s \ K ( z ,w)\ < sup zer \ s K(z,z). 

In view of Lemma 4.1 we have 5s C GExp(X) Exp(iG)G for some X e iC . Therefore we get 
for all s G S that 

\\K rsS0 \\ 2 = K(r S s a ,r S s ) = ||^( SSo )^ r |i 2 < sup geG ||7r(Exp(X))7r( 3 ).w r || 2 , 

since (ir,H) is contractive and Exp(X) commutes with Exp(iG). But now the assertion follows 
from Lemma 4.2(h). 

(ii) Let / G . Then / G w(so)-'H for some s = Exp(X ) , X G iC . Thus we can write 
/ = w(so).g with g G TL. But then (i) implies that 



sup ses |/(rs)| = su PseSso \g(s)\ = su PseSso \{g,K r ,)\ < C(a ) \\g\\. 
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Vanishing at infinity 

In this subsection we show that the analytic vectors of certain invariant Hilbert spaces of 
holomorphic functions on T\S are vanishing at infinity. 

Lemma 4.5. Let s = Exp(X ) G S, X G iC , and (s„)„ £ n be a sequence in Ss . 

(i) There exists an element X G iC such that 

s n = QnK 1 Exp(Jf + X n )h n 

with g n , h n e G, X n e iC. 

(ii) Assume now that T < G is cocompact and that (rs„)„ eN leaves every compact subset in 
T\S . Then the sequence h~ l Exp(X + X n )h n leaves every compact subset o/ Exp(VF) . In 
particular we have X n — > oo in iC or h n — > oo in G. 

Proof. (i) This follows from Lemma 4.1. 

(ii) Note that there is a polar decomposition of Y\S 

T\G x W -> T\S 

which is a homeomorphic mapping. As T\G is compact, (ii) follows now from (i). ■ 

Theorem 4.6. (Vanishing at Infinity) Let T < G be cocompact and (ir\,H\) be a T -spherical 
unitary highest weight representation of G with A G i int C* . Assume further that G is a ( CA )- 
Lie group and that Z(G) is compact. Let v G H 1 ^ and f v (Ts) = (Tr(s).v,v r ) the corresponding 
holomorphic function on T\S . Then f v extends to a continuous function on T\S , also denoted 
by f v and the following assertions hold: 

(i) The function f v is bounded. 

(ii) We have lim s^=o_ /„(s) = 0. 

Proof. Fix X S iC and set s t : = Exp(tX ) for t > 0. Then W{ = U t>0 7r A (s t ).HA shows 
that there is a to > such that v = TT\(s to ).w for some w G TL\ . Set s n : = s to ■ Then Sso C S 
shows that there is a continuous extension of /„ to T\S . 

(i) Proposition 4.4(h). 

(ii) Let (s„)„ e N be a sequence in S such that Ts n — > oo in T\S . We claim that rs„s — > oo in 
F\S . For that it is sufficient to show that the mapping 

pr(s y.T\S ^T\ T Ts^Tso 

is proper. Recall from [HiNe93, Th. 3.20] that S acts via right multiplication on S by proper 
maps. In particular, p(s ): S — > S, s ss is proper. Let now Qr C T\5 be a compact 
subset. Then we find a compact subset Q C S such that Qr = r\TQ. Since p(so) is proper, 
the set p(so) _1 (Q) is compact. Hence pr(so)^ 1 (Qr) = r\rj°( s o) _1 (Q) is compact, establishing 
our claim. 

By Lemma 4.5 we find an element IgC such that 

s n s = gnh^ 1 Exp(X + X n )h n 
for elements g n ,h n G G , X„ G iC. Then Lemma 4.3 and Proposition 4.4 show that 
f v (Ts n ) = f w (Ts n s ) = (w,K rSn s„) = (w,7r A (s s*).w r ) 
= (w^xih- 1 Exp(X + X n ))^{h n g^).v T ) 

= {-K X {h n ).w,-K X {Vx. V {\x + X„))7r-"(Exp(ix)M- 1 ).i; r ). 
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Since T\G is compact, Lemma 4.2(ii) implies that Q: = n x ~ u) (~Exp(^X)G).v r is compact in TL\. 
Thus we get 

(4.2) \f v (Ts n )\ < sup ueQ \(7r x (h n ).w,TT X (Exp(^X + X n )).u)\. 

As Ts n tends to infinity in T\S we have h n — > oo in G or X n — > oo in C (cf. Lemma 4.5(h)). 
Case 1: X n — ► oo. Then |X + X„ — > oo in iC and we have 

||7r A (Exp(ix + X n ))\\ = sn PweWt e -*(**+*») 

(cf. [Kr99, Lemma 4.13]). Thus A G iintC* and the fact that C is >V{ -invariant (and hence 
iintC*) shows that ||7r A (Exp(±A + X n ))\\ -> 0. It follows from (4.2) that 

hm \f v (Ts n )\ < \\w\\ ■ su PueQ \\u\\ ■ lim \\tt x (Ex P (1-X + X n ))\\ = 0. 

Case 2: /i„ — » oo. We may assume that X„ stays in a compact subset of iC because of Case 1. 
Then 7r A (Exp(iA + X n )).Q stays in a compact subset Q 1 D Q of W A and (4.2) implies that 

|/„(rs„)| < sup ueQ , \(irx(h n ).w,u}\. 

Note that (ir\,H\) has compact kernel, since G is a (CA)Lie group, Z(G) is compact and 
A G iintC* (cf. the proof of [Kr98a, Prop. V.7]). Hence the facts that (ir\,H.\) is irreducible 
and G is a (CA)-Lie group imply that Mayer's generalization of the Howe-Moore Theorem (cf. 
[Ma97, Prop. 3.4], [HoMo79, Th. 5.1]) applies and so lim,^^ |/(rs n )| =0. ■ 



Separating holomorphic functions 

Now we are ready to prove the main result in this section. Recall from [Ko98, Ch. 3, §2] 
the definition of the Caratheodory semimetric and note that the Caratheodory semimetric is a 
metric if and only if the bounded holomorphic functions separate the points. 

Theorem 4.7. (Separation of Points) Let S be a complex Ol 'shanskii semigroup associated 
to a ( CA )-Lie group G . Let T < G be an arbitrary discrete subgroup of G . Then the following 
assertions hold: 

(i) For every zi,z 2 G T\S , Z\ ^ z 2 , there exists an f G S 1 (r\S l ) such that ^ f(zi) ^ f(z 2 ) ■ 
In particular Hol(r\S f ) separates the points of T\S . 

(ii) IfT<G is cocompact, then the bounded holomorphic functions on T\S separate the points. 
In particular, the Caratheodory semimetric on T\S is a metric. 

Proof. (i) The proof uses a trick which was used by Siegel in order to show that the Siegel 
modular forms separate the points on G/K (cf. [Fr83]). 

Let Co(r\5) denote the algebra of continuous function on T\S which vanish at infinity. 
Then it follows from [Kr98a, Prop. II.4(i)(b)] and a slight adapation of [Kr98a, Lemma V.13] 
that B 1 (S) n Co(S) separates the points of S . 

Let now Ts 1 , Ts 2 G T\S be two different points. Let / G B 1 (S) n C (S) be such that 



m: = sup 7er {|/(7Si)|, \f(^s 2 )\} > 0. 
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By Proposition 3.4 we know that for every / G B 1 (S) and s e S one has 
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(4-3) |/ r ( S )|<^|/( 7S )|<^ 
and f G B 1 (r\S'). 

Hence the supremum m is actually attained and without loss of generality we may assume 
that m = 1 = /(si) . Let now 

r„:= {7 er\{l}: 1/(7*01 = 1 or |/( 7 s 2 )| = 1} 

and note that T is a finite set by (4.3). Since B 1 (S) (~l Co (5) separates the points of S we find 
age B^S) n C Q (S) such that 

ff (s0 = 1 and ff ( 7 s0 - ff(7s 2 ) = (V 7 G T ). 

For every n £ N we now consider the function 

F n (s)=g(s)r(s) 

and note that F„ G -B 1 (S 1 ) . Averaging over T we obtain that 

(F n f(T S ) = J29hs)r( 1S ) 

for all s G S. Note that (F„) r G B 1 ( r \ 5 ') • Now we have 

i(F™) r i(r S2 )< X! i»(7«2)iir(7«2)i 

7er\r 

and we see that lim„^ oc (F„) r (rs 2 ) = by (4.3) and |/(7S 2 )| < 1 for all 7 G T\T . On the 
other hand we have 



(F n ) r (r Sl )=g( Sl )f( Sl )+ ]T g( 1 s 2 )f n ( 1 s 2 ) = l+ ^ 9(1*2). f n (1*2) 

7er\r 7er\r Q 

and by the same reason as before we have lim„^ 00 (i 7 ' ri ) r (rsi) = 1. Hence there exists an n G N 
such that {F n ) r (T Sl ) ^ {F n ) r (Ts 2 ) proving (i) 

(ii) In view of the proof of (i), this follows from Proposition 4.4. ■ 

Remark 4.8. By Neeb's version of the Gelfand-Raikov theorem (cf. [Ne99, Th. XI. 5.1]) for 
complex Ol'shanskh semigroups, the bounded holomorphic functions on S separate the points. 
It follows that S is hyperbolic complex. Hence T\S is hyperbolic complex for arbitrary discrete 
subgroups r < G by [Sh92, §20, Th. 1]. To compare this with the results above notice that 
Theorem 4.7(h) is a stronger statement than that Y\S is hyperbolic complex. ■ 
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Example 4.9. (a) Let G be a linear Hcrmitian Lie group. Then by a theorem of Borcl (cf. 
[Bo63, Th. C], [Ra72, Th. 14.1]) there exist cocompact lattices T < G. Further the Lie algebra 
of q admits pointed open convex Ad(G) -invariant elliptic cones ^ W C g. Set S = Tg(W). 
Then by Theorem 4.8 the bounded holomorphic functions on T\S separate points. 

(b) Let f)i — RX © RY © RZ be the 3-dimensional Heisenberg algebra with relations [X, Y] = Z 
and all other brackets vanishing. Then o = RT acts on \) by derivations via [T, X] = Y , 
[T, Y] = —X and [T, Z] = . Then g = f) x a is a four dimensional solvable Lie algebra, the 
so-called oscillator algebra. It is the basic example of a solvable Lie algebra admitting pointed 
open invariant convex cones ^ W C q. Note that t = IZffio is a compactly embedded Cartan 
subalgebra of q . 

Let H\ = R 3 denote the simply connected Heisenberg group corresponding to f)i . We have 
a connected Lie group G = H\ x S 1 with Lie algebra fl. Then L = Z 2 x ^Z C Hi is a cocompact 
lattice in G and hence the bounded holomorphic functions on T\S separate points by Theorem 
4.8. 

(c) Now we give an example of a Lie group G of mixed type, i.e., G is neither reductive nor 
solvable. We define a quadratic form 

F:R 3 ^R, (x, y, z) ^ x 2 + y 2 - 3z 2 

and let L: = SOq(F, R) denote the connected component of the special invariance group of F. 
Then F does not represent zero in a non-trivial rational way so that r^:= SOo(F, Z) forms a 
cocompact lattice in L (cf. [Bo69, 8.6, Ex. 1]). 

Let (-, -}2.i denote the non degenerate symmetric bilinear form on R 3 induced by F . Then 
the prescription 

n F : (R 3 x R 3 ) x (R 3 x R 3 ) -> R, ((u, v), (u , v')) ^ (u, v') 2 ,i - (u , v) 2 ,i 

defines an L -invariant symplectic form on R 3 x R 3 (we let L act diagonally). Associated to Of? 
we build the nilpotent Lie algebra u: = (R 3 x R 3 ) x R with bracket 

[((u,v),z),((u',v'),z')} = (0,Q F ((u,v),(u',v'))). 

Note that u is isomorphic to the 7-dimensional Heisenberg algebra ()3 . Let U denote a simply 
connected Lie group with Lie algebra u. Then U = (R 3 x R 3 ) x R and Ty: = (Z 3 x Z 3 ) x |Z is 
a cocompact lattice in U which is stable under T L . Thus r: = Tjj x T L is a cocompact lattice 
in the semidirect product G: = U x L. 

Note that there is an embedding of q into the Jacobi algebra f) 3 x sp(3,R) sending u 
isomorphically onto f}3 and I: = L(L) into Sp(3, R) . In view of [Nc99a, Ch. VII], this embedding 
guarantees us the existence of non-trivial pointed Ad(G) -invariant open convex cones W C q. 
Now the bounded holomorphic functions on L\L G (M / ) separate points by Theorem 4.8. ■ 



5. The Stein property of T\S for T\G = S1(2,Z)\ S1(2,K) 

We let the group G = S1(2,R) act on the upper halfplanc n+:= {z 6 C: lmz > 0} by 
means of Mobius transformations: 

az + b ( a b\ „ „. 

g.z = — g = I , G G, z G 11+ 



cz + d \c d 

Note that Gc = Sl(2, C) . The compression semigroup of II + 

S:={geG c :g.n+ CU+} 
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defines a closed complex CTshanskii semigroup with interior 

S:={gEG c :g.TFcn+}. 

Let U = ( ^ ^ ) be the infinitesimal generator of the maximal compact subgroup K = 



-1 0^ 

SO(2,R) of^G. Set'lR" =] - oo,0[. Then it is easily shown that 

S = Gexp(iR~U)G = Gcxp(iW) 

with W = Ad(G).R"£/ the lower light cone in g = sl{2, R) . Note that W and -W are the only 
non-trivial Ad(G) -invariant pointed convex cones in g. 

Let T = Sl(2, Z) be the modular group. For z E II + set q: = e 2nlz . Then the discriminant 

oo 

A:n+^C, z^qY[(l-q n ) 24 

n=l 

defines a modular form of weight 12, i.e., A is holomorphic, vanishes at infinity and satisfies 

(c 7 z + rf 7 )- 12 A( 7 .z) = A(z) * e n+ )7 =(£ ^ 

a \ | G G and z e II + we define 
c a J 

/j,(g, z): — cz + d 



and note that n is a cocycle, i.e., /j,(gig2,z) = n(gi, g2-z)p,(gz, z) for all g\,gi, € G, z E n + . 
Then the description 

/ A :G^C, 5 ^ M ( 5 ,z) 12 A( 5 .z) 

defines a left T -invariant analytic function on G (cf. [Bo66]). Since S compresses II + , we see 
that [i extends to a continuous function fi(s,z), s E S, z E 11+ , with no zeros. By the same 
reason the prescription s A(s.i) defines a continuous function on S . Hence we can analytically 
continue /a to a continuous function 

F A :S^C, s /i(s, i) 12 A(s.i) 

which is holomorphic when restricted to S . Note that Fa is left T -invariant and hence factors 
to a function on T\S , also denoted by Fa . 

It is a consequence of [KO01, Th. 3.2] that Fa vanishes at infinity on T\S , i.e., 

(5.1) ^irn^ F A (rs) = 0. 

r»er\s 

Let us briefly scetch the proof of this fact. The discriminant A is bounded on a fundamental 
domain for F in n + . Hence it is sufficient to show that lim=^~ /u(s, i) = 0. For G = S1(2,K) this 
can be seen by direct calculation; more generally it follows from the fact that the middle projection 
k: P~ KcP + — ► Kc restricted to S is a proper map and has image k(S) = Kcxp(iM.^U) (cf. 
[KO01, Prop. 1.2, Cor. 2.4]). We note that Fa has no zeroes on T\S . Hence jp^j defines a 

continuous plurisubharmonic function on T\S and (5.1) implies that 
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Theorem 5.1. Let S be a complex 01 'shanski semigroup associated to G = S1(2,R) and let 
r = Sl(2, Z) . Then T\S is Stein. 

Proof. In view of Grauert's solution of the Levi problem (cf. Theorem 2.2), it is sufficient to 
prove the existence of a strictly plurisubharmonic exhaustion function of T\S . 

Let ip be the non-negative biinvariant plurisubharmonic function on S from Theorem 2.4. 
Then tp factorizes to a function on T\S which we also denote by ip . Then 

1 

tp: = ip + 



\Fa\ 



defines a strictly plurisubharmonic function on T\S . To conclude the proof of the theorem it 
suffices to show that <p is proper. Let (z„) nS N be a sequence in T\S leaving every compact 
subset of T\S . Then we either have z„ — > z e (r\<S) — (T\S) or z n — > oo in T\S . In the first 
case we have ip(z n ) — > oo by Theorem 2.4 while in the latter case linin^oo \ F ^ Z jj = oo by (5.2). 
This concludes the proof of the theorem. ■ 

Remark 5.2. (a) Note that by Theorem 2.3 now also for all subgroups Lo < Sl(2, Z) the 
manifold Toys' is Stein. 

(b) Let G be a hermitian Lie group and L < G a lattice. We claim that there exists no 
holomorphic cusp forms on G/K without zeros if L < G is cocompact or if the real rank 
rank(G) of G is strictly larger than one. Since if / were a holomorphic cusp form without zeros, 
then j would again be an automorphic form. If T < G is cocompact, this is clear, and if T < G 
is not cocompact and rank(G) > 1, then this is implied by the Koecher-principle (cf. [Fr83, 
Ch. I, Hilfssatz 3.5]). Now j would be of "negative weight" which is impossible since there are 
no automorphic forms of negative weight (if T < G is cocompact, this is a consequence of the 
Plancherel Theorem for L 2 (T\G) and in the non-cocompact case this can be proved as in [Fr83, 
Ch. I, Satz 3.13]). 

Therefore our construction for S1(2,R) cannot be generalized to arbitrary hermitian groups 
nor to the case of cocompact subgroups. ■ 

Conjecture 5.3. We conjecture that T\S is Stein for all complex CTshanskii semigroups and 
all discrete subgroups T < G . In view of the proof of Theorem 5.1, it would be sufficient to find 
a holomorphic function on T\S with no zeros and which vanishes at infinity on T\S . If T < G 
is cocompact, then we have seen in Theorem 4.6 that all matrix coefficients f v (Ts) — (tt(s).v, v r ) 
for v E TL U vanish at infinity. The difficulty is to show that there exists a certain v <G Ti u so that 
/„ has no zeros. As we pointed out in Remark 5.2, this can never happen if v is a highest weight 
vector. But it is likely to be true that /„ has no zeros for all v G n\(S).v T , as the reproducing 
kernel 

K(Ts,Tt) = (Tr(t*).v r ,ir(s*).v r ) 

of the realiztion of (tt,H) in Howrys') should be zero-free by general principles. However, this 
seems to be a very challenging problem, even for G = Sl(2, R) . ■ 
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